数据仓库建模_5

Definition  of technical architecture

The technical architecture is the overall plan for what you want the DW/BI system to be when it's ready for serious use .
 It describes the flow of data from the source systems to the decision makers and the transformations and data stores that data goes through along the way. It also specifies the tools, techniques, utilities, and platforms needed to make that flow happen.

We think about metadata as all the information that defines and describes the structures, operations, and contents of the DW/BI system.The DW/BI industry often refers to two main categories of metadata: technical and business.

Technical metadata defines the objects and processes that make up the DW/BI system from a technical perspective. This includes the system metadata that defines the data structures themselves, like tables, fields, data types, indexes, and partitions in the relational engine, and databases, dimensions, measures, and data mining models. In the ETL process, technical metadata defines the sources and targets for a particular task, the transformations (including business rules and data quality screens), and their frequency.

Process metadata describes the results of various operations in the warehouse. In the ETL process, each task logs key data about its execution, such as start time, end time, CPU seconds used, disk reads, disk writes, and rows processed. Similar process metadata is generated when users query the warehouse. This data is initially valuable for troubleshooting the ETL or query process.

Note of use delete in kettle

Since in the "delete" the "=" condition  cannot deal with null = null, So if some columns in the comparison are null,the "delete" will be unable to deal with it.

最可悲的是什么

是你当初喜欢的不染人间烟尘的女子变成了一个菜市场的大妈.

在汇款小票丢失以后.

作为一个成年人,遗失汇款小票是一件可耻的事情,但是它发生,总得解决吧.
首先得明确这个汇款小票是无法补开的,但是可以补开一个凭证,其实是一个凭证的复印件.
需要的信息,1,汇款行2,日期(到天就可以了)3,汇款对象的信息,帐号,账户名,开户行4金额5柜员号.
可能对于大家来说柜员号的话是很难得知的,不要紧.如果你记得是那一个窗口,可以找大堂经理查询,如果不记得了也没有关系,可以问当日所有的柜员号信息.(银行内部其实是可以根据对方账户信息查到的,我就是这样查到的,但应该不符合规范)
一,找到这笔业务归属的支行网点,这个可以问大堂经理知道.
二,找到该支行的大堂经理,向他说明情况,他们银行应该是提供这种业务的,然后他会带你去开这个复印件.
三,找到这个人以后,把上面的信息给他,他就可以给查了,但是这个查询是要付费的(我明天才去,还不知道多少钱),如果无法确定是哪一个柜员号,至少把当天的当班的所有柜员号给他,这样可以缩小范围,而且,这种情况下,由于业务量比较大,可能需要说说好话.
这样应该就可以搞定了.
Note,好像现在大堂经理都还比较负责,可能是比较害怕投诉吧.当然最好还是不要撕破脸皮,这样他可以提供规章之外的一些帮助.否则按规章的话是很难办好这件事情的.
尽量让对方多解释,这样的话有时候对方可以提供一些有用的信息,比如我去咨询的时候大堂经理就说如果提供窗口号可以知道是那一个柜员,而窗口是不会变的,这样我们就可以推导出他们是可以提供当天所有的柜员号.
嗯,当然我的例子还只是一个半成功的例子.明天上午或者下午就去执行第三步了.
To be continue.

H1N1 continue.

Turns out that It was not H1N1 flu but some other respiratory disease. Or
so called bronchitis. Lucky but not most lucky of me .

H1N1

咳嗽(以前感冒只会流鼻涕),腹泻,头微疼,腹泻过一次.
可能是H1N1了,也可能是回北京风吹得太过,着凉了.郁闷啊.

微软拼音输入法引起的蓝屏问题

key words Windows XP 输入法 无法进入系统 蓝屏 默认输入法 修改 input method blue screen modify 

之前一直用的是Google输入法,Dvorak 键盘布局,昨天有人想用本,装上了微软拼音输入法2003, 并设置为默认输入法,然后出现了无法进入系统的症状,当进入系统的时候winlogon和explorer进程都统统死掉了.安全模式也是一样.然后出现蓝屏00021a hard error.

万幸我还另外装了一个linux系统fedora(之前就已经拯救我于北信源XX网络监视器),搜索了一下,解决方案如下.

安装注册表编辑器chntpw, yum上面有. cd 到%UserProfile%例如C:\Documents and Settings\Administrator下,

chntpw NTUSER.dat

cd Keyboard Layout 

cd Preload

编辑 1 (输入?获得帮助)

改为 00000804 US-en,这个应该没有问题.

退出,现在问题就解决了.

I added a new input method to my Input Method, and set it to be the default input method, terrible thing happened, I can't log into Windows XP any more.It must be some fatal error occurs when the new default input method loads. Luckily I got my Linux OS Fedora.

Following is how the problem is settled.

log in Fedora , install chntpw by yum.

cd to %UserProfile% eg C:\Documents and Settings\Administrator.

chntpw NTUSER.dat

cd Keyboard Layout

cd Preload

'edit' 1 (sorry for forgetting the command,use ? to get help)

change to to 00000804 Us_en.

q
I believe the problem is no longer a pain to you, Wish it helps.

数据仓库建模_4

return on investment (ROI)
net present value (NPV) or internal rate of return (IRR)
The major roles involved in the DW/BI implementation front office, coaches, regular lineup, and special teams

Front Office: Sponsors and Drivers Folks in the front office aren't involved on a daily basis, but they have a significant
influence on the rest of the team.

DW/BI Director/Program Manager

Some team members are assigned to the project full time; others are involved on a
part-time or sporadic basis.

Project managers need strong
organizational and leadership skills to keep everyone involved moving in the same direction.

Business Analyst
The business analyst is responsible for leading the business requirements definition activities and then representing those requirements as the technical architecture,

Metadata Manager

This person is not responsible for inputting all the metadata, but rather is a watchdog to ensure that everyone is contributing their relevant piece,

Special Teams

Technical Architect/Technical Support Specialist Depending on your environment, there may be platform specialists who get involved in early stages of the design to perform resource and capacity planning.

Lead Tester he lead tester develops test methodologies, and confirms that the tests cover the span of the data, processes, and systems.

The DW/BI project needs a detailed, integrated project plan given its complexity, both in terms of tasks and players.

If a single task requires more than two weeks, subtasks should be identified. 我觉得超过1周的都需要.

best project management software package will be worthless unless the time is invested to input and maintain the project plan

tracking the following information for each task

Resources: 

Original estimated effort

Original estimated start date

Original estimated completion date:

Status:

Updated start date

Updated completion date:

Effort to finish:

Late days:

% Completed:

Dependencies:

Also don't forget that ETL development is a classic software development task with five distinct phases: development, unit testing, system testing, acceptance testing, and final rollout with documentation. Inexperienced project managers may focus most of their attention on what turns out to be only the first of the five steps. ETL的测试需要分层级.

数据仓库建模_3

In reality, scope and justification go hand-in-hand,
划定范围和调整是同时进行的.
Note Project scope should be driven by the business's requirements.
范围是由业务需求决定的.
Each early iteration of your DW/BI program should be limited in scope to
the data resulting from a single business
process.起初的时候不能简单问题复杂化.In other words, start small

Given the lengthy project timeframes associated with complex software
development projects combined with the realities of relatively high rates
of under-delivery, there's
been much interest in the rapid application development movement, often
measured in weeks. 快速开发(通常以周记)

In fact, we suggest that BI team members work in close proximity to the
business so they're readily available and responsive;
开发人员和业务人员更亲近,业务为导向的.

Some development teams have naturally fallen into the trap of creating
analytic or reporting solutions in a vacuum. In most of these situations,
the team worked with a small set of users to extract a limited set of
source data and make it available to solve their unique problems. The
outcome is often a stand-alone data stovepipe that can't be leveraged by
others or worse yet, delivers data that doesn't tie to the organization's
other analytic information.
做分析做成了报表,这不正是我们组现在的问题所在么?因为孤立的获取需求,所以做出来的东西也一个个的孤立了,不能互相使用.

project charter, the document explains the project's focus and motivating
business requirements, objectives, approach,anticipated data and target
users, involved parties and stakeholders, success criteria,assumptions and
risks. It may also be appropriate to explicitly list data and analyses
关注点,需求,客观现状,途径,预计中的数据和目标用户,相关的组织\股东,成功标准(验收标准),前提和风险

P38

weired way to get return value

I guess shell is always weired. if you run a command in a subroutine in a
script, try return in the command field and then return $? in the
subroutine field. as what following shows .

function isRunnable()
{
typ=$1;
filename=$2;
day_a=$3;
day=${day_a:=''}
cat 'exe_seq_all.dat' | grep "$filename$day$" | while read line
do
arr=($line);
prework=${arr[0]};
if [ ! `grep -c " $prework " 'fin.dat' ` = 0 ]
then
return 0;
fi;
return 1;
done
return $?;

recover from rm *

I had a good meal after hard afternoon work. when i came back, I typed rm
* in the cygwin command line. all the work I've done in the afternoon is
gone, Much upset when I learn there is no 'recycle bin' in cygwin.
lucky, after some search on the net , I found this 'File Scavenger' which
help me to restore the *.sh *.dat *.pl *~ from the disk. Thank Que Tek
Consulting.co

note from the data warehouse lifecircle toolkit

Three keys to ensure BI/DW success
1 Strong Senior Business Management Sponsor(s)
2 Compelling Business Motivation
3 Feasibility

DW/BI systems that align with strategic business motivations and
initiatives stand a
good chance of succeeding.

In some sense, any SELECT DISTINCT investigative query on a database
field could be considered data profiling

Isolation suitable for ETL

The best isolation level for ETL is DIRTY READ, Cauz it wont set any locks on anything , and wont consider lock exists on any anything , as we have a great chance to ETL DATA from a static data base, As a result, It will cost the least and execute in the fastest way.

In informix, the clause is SET ISOLATION TO DIRTY READ.

the following is the isolation performance in ORACLE. 

Read uncommitted is the same as DIRTY READ.

                  Dirty Read     NonRepeatabe Read       Phantom Read 
Read uncommitted  Possible       Possible                Possible
Read committed    not possible   Possible                Possible
Repeatabe read    not possible   not possible            Possible
Seriaizabe        not possible   not possible            not possible

change is difficult

I used to confused by the fact people are always not listening, And now I
have exprienced the same things, though it's me that not listening. Why is
that? lack of self-confidence?
Maybe,

自己在做什么

看到空间里面同学已经在暗自下决心要做hw的对手了.自己在做什么呢?

kill the current proccess in dbaccess on unix

try ctrl + \

昨夜遭遇

昨天下班回去,一个人乘电梯回房间,先看到电梯从二楼下来.门开了,里面有一个女的,装扮还挺正常,样子也还行,然后很奇怪的没有下电梯(电梯最低1层),我还以为她是要上去,还问她,'要上么?'然后那女的也没有怎么回答,然后我到三楼,她到五楼.然后我就转过身,感觉那个女的一直盯着我看,到二楼了,那女的突然问,你住那个房间.我:'呃~~嗯~~323'.然后那女的又问,是一个人么,可怜我还没有意识到这个女的是干什么的,心中暗感诡异,此时电梯也到三层了,我一边疾出电梯,一边脱口而出,'不是'.走出3,5米远之后我,我突然开始怀疑了.不会吧,我难道碰上了妓女?我已经到了这个年纪了么?会被妓女当成潜在的顾客.悲剧啊,可怜我单身这么多年了.

cognos sdk dev note

get all the members of an role
the search path is like "expandMembers(CAMID(\"::System Administrators\"))" 

got all the roles under certain namespace "CAMID(\":\")//role"

got the object the querier has read or traverse permission //folder[permission('read') or permission('traverse')]

get all the groups and role a user or set of user belongs to "membership(CAMID(\"dbAuth:u:8888888888\"))"

遗失了什么?

郁闷,今天升级opera 10, 把我之前写的一封信的草稿弄没了,虽然一直放在那个地方没有胆敢发出去,但是想要留在那里也许有一天自己回去看.

难道这其实是天意?让我不要纠缠在那里.

cognos sdk note

when use the search capability of cmservice in sdk, you can traverse the
name even if you dont have the permission to access that, although you
could not browse it.

The implements of the service interfaces are often ended with 'stub'.

/directory is the path of namespace .

use query(searchPath, properties, sortBy, options)
properties is the enumerator of the properties you want to get.

"execute, read, setPolicy, traverse, write" are the five permission of
NmtokenArrayProp which contain the list of the privaleges.